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Abstract (195 words) 

Wearing face masks is one of the essential means to prevent the transmission of certain 
respiratory diseases such as COVID-19. Although acceptance of such masks increases in 
the Western hemisphere, many people feel that social interaction is affected by wearing a 
mask. In the present experiment, we tested the impact of face masks on the readability of 
emotions. The participants (V=41, calculated by an a priori power test; random sample; 
healthy persons of different ages, 18-87 years) assessed the emotional expressions 
displayed by twelve different faces. Each face was randomly presented with six different 
expressions (angry, disgusted, fearful, happy, neutral, sad) while being fully visible or 
partly covered by a face mask. Lower accuracy and lower confidence in one’s own 
assessment of the displayed emotions indicate that emotional reading was strongly 
irritated by the presence of a mask. We further detected specific confusion patterns, 
mostly pronounced in the case of misinterpreting disgusted faces as being angry plus 
assessing many other emotions (e.g. happy, sad and angry) as neutral. We discuss 
compensatory actions that can keep social interaction effective (e.g. body language, 
gesture and verbal communication), even when relevant visual information is crucially 


reduced. 
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Main text 

Wearing face masks! is indicated in many scenarios, mostly in clinical contexts, when being 
infected by certain respiratory diseases or in times of epidemics where the risk of potential 
transmission through air passages has to be reduced (Jefferson et al., 2008). During the COVID- 
19 pandemic (Coronavirus disease 2019), most countries and health organizations like the WHO 
propagated wearing face masks by early 2020 as a key strategy to reduce the spread of the SARS 
2 (Severe Acute Respiratory Syndrome) coronavirus. 

Face masks do not only have a direct positive medical impact in terms of preventing the 
virus from spreading to those who are most vulnerable (Wu & McGoogan, 2020); they also have 
positive societal effects as wearing masks allows for relaxing other preventive measures such as 
strict isolation and quarantining (Mniszewski, Del Valle, Priedhorsky, Hyman, & Hickman, 
2014). However, face masks also cover, per definition, a major part of the human face, which 
can crucially affect social interaction. Our faces provide the key information of “person 
identity”, “directed visual information” (e.g. attractiveness, age, sex), information supporting the 
understanding of speech by enabling “facial speech analysis” as well as fine-grained information 
that allows for reading the other’s emotional state via “expression analysis” (Bruce & Young, 
1986). We can compensate a lack of signal for all of these facets of face processing (Griiter & 
Carbon, 2010), but often we might reduce the efficacy of processing, the confidence in our 
assessments and we are susceptible to lose a part of the multichannel-multisensory integration 
possibilities to cross-check and validate our assessments. Some of these signals that faces 


provide are processed very fast (identity, Carbon, 2011; gender and attractiveness, Carbon, 





' Face masks show a great variety of forms and technologies; within the present paper, we will focus on masks that 
look like simple surgical masks and that people can fabricate themselves, so called community masks. 
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Faerber, Augustin, Mitterer, & Hutzler, 2018; emotion, Willis & Todorov, 2006), although the 
validity of the final assessments are under great dispute (Rojahn, Gerhards, Matlock, & Kroeger, 
2000; Russell, 1994). 

With regard to expression analysis, different studies showed that we are far from being 
perfect in assessing the emotional state of our counterpart just by inspecting the face (Derntl, 
Seidel, Kainz, & Carbon, 2009) without knowing the context of a scene (Aviezer et al., 2008) or 
without information about the dynamic evolvement of the seen expression (Bassili, 1979; Blais, 
Fiset, Roy, Saumure Régimbald, & Gosselin, 2017). A partial occlusion of the face (Bassili, 
1979), e.g. by sunglasses (Roberson, Kikutani, Doge, Whitaker, & Majid, 2012) or by scarfs 
(Kret & de Gelder, 2012)is a further obstacle to accurately reading emotions from facial 
expressions (Bassili, 1979). Face masks or community masks, as the ones commonly worn 
during the COVID-19 pandemic to shield the mouth and the nose, cover about 60-70 % of the 
area of the face that is relevant for emotional expression and thus emotion reading (e.g. ~65% in 
the case of the depicted persons in our face set—exact numbers are hard to tell; we can only rely 
on rough estimations as indicative face areas differ across persons). Crucially, these masks cover 
an area of the face that is crucial for the effective nonverbal communication of emotional states. 
Although specific research on the impact of such face masks on emotional recognition is 
missing, there are some indications from research on the effect of different kinds of facial 
occlusions. An important source of data are the so-called “Bubbles”-experiments that make use 
of a general technique developed by Gosselin and Schyns (2001). This technique allows for 
identifying the specific visual information that is most relevant to human categorization 
performance, for instance information needed to express and read emotions. Other paradigms 


comprise the presentation of top vs. bottom halves of faces (Bassili, 1979) or the partial 
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occlusion of target faces with ecological valid items such as a niqab (Fischer, Gillebaart, 
Rotteveel, Becker, & Vliek, 2012), a shawl or a cap (Kret & de Gelder, 2012) in order to test for 
differences in the participants’ emotion reading performance. These different paradigms operate 
with very different stimuli and they were used with samples from different populations. Taken 
together, they do not enable immediate conclusions about the specific impact of face masks on 
the reading of different emotions. The manipulations realized in those paradigms are, neither 
quantitatively nor qualitatively, analogous to the actual practical use of face masks. Further, the 
results of studies that applied these paradigms are often incoherent, sometimes even 
contradictive. There is, for instance, a relatively high consensus that covering the lower face 
parts yields reduced performance in assessing a happy emotional state (e.g., Eisenbarth & 
Alpers, 2011; Fischer et al., 2012; Kotsia, Buciu, & Pitas, 2008). For other emotional states, 
however, there are quite contradictive results to be found in the literature, e.g., for fear detection 
(in favour of higher relevance of the eyes, see Bombari et al., 2013; in favour of higher relevance 
of the mouth, see Kotsia et al., 2008). There is even evidence that a partial coverage of the face 
might lead to better performance due to fading out irrelevant or deceptive information in faces 
(Kret & de Gelder, 2012). Laypersons, for instance, were more accurate in detecting deception in 
persons who wore a niqab than in persons who did not (Leach et al., 2016). Inconsistent results 
such as angry faces attracting more attention to the eyes than the mouth (Eisenbarth & Alpers, 
2011) while the occlusion of the mouth resulted in lower accuracy of detecting anger (Kotsia et 
al., 2008) have to be interpreted with caution as we do not know the causal or temporal 
interdependence of such processes. Specific types of occlusions might interfere with different 
emotions: For example, the mouth seems important for the detection of happiness and fear, but 


the eyes are more relevant for anger, fear, and sadness (Bombari et al., 2013). 
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The present study specifically tested how a common face mask, which for instance 
dominates the social scenes during the COVID-19 pandemic, changes the efficacy of emotion 
reading expressions displayed by different faces. Besides recognition sensitivity, we were 
particularly interested in the confusion of certain emotions with other emotional states due to an 
increase of signal ambiguity in order to understand everyday life problems in effectively 


communicating when wearing face masks. 


Experimental Study 
Methods 
Participants. The needed sample size of N = 36 was calculated a priori via power analysis (Faul, 
Erdfelder, Lang, & Buchner, 2007) targeting a repeated measures Analysis of variance 
(ANOVA) with 6 groups (emotions) and 2 measurements (mask vs. no mask) and the ability to 
detect a medium effect size of f= 0.25 (Cohen, 1988), given an a = 0.05 and a test power (1-f) = 
0.80. From our entire set of data from 41 participants (Mage = 26.7 years [18-87 years], Nfemale = 
30) we could use all data sets as all participants reached the pre-defined criterion of showing at 
least a performance of correctly identifying emotional states in 50% of the cases where faces 
were presented without masks (actually, the performance was much higher, see results). This 
slightly higher actual than needed number of participants resulted in an achieved post hoc test 
power of 0.88. 
Material. All face stimuli were obtained from the MPI FACES database (Ebner, Riediger, & 


Lindenberger, 2010) by a study-specific contract effective by 27 April 2020. As base faces on 
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which we later applied face masks, we used frontal photos of 12 Caucasians (6 female, 6 male) 
who belonged to three different face age groups (young, medium, elderly) yielding two persons 
per face sex x face age group cell. For each person, six different pictures were used that showed 
the emotional states angry, disgusted, fearful, happy, neutral and sad. For the application of face 
masks to all of these 72 original pictures we photographed a typical homemade (beige) 
community mask. The image of the mask was cut out via Photoshop and individually applied to 
the different face versions. Realistic shadows were added to create maximally realistic and 


plastic pictures of persons wearing a face mask (Figure 1). 


angry disgusted fearful happy neutral sad 





Figure 1: A person showing six different emotions without a mask (A) and wearing a mask (B). Original material from top row 
stems from MPI FACES database (Ebner et al., 2010). 


In sum, we obtained 2 [face sex] x 3 [face age group] x 2 [individuals] x 6 [emotions] x 2 [no 


face mask vs. face mask] = 144 face stimuli. 


Procedure. The experiment which ran on the SoSciSurvey online platform was conducted 


between 15 May (10:01 local time) and 18 May (19:45 local time) during the COVID-19 
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pandemic when general legal obligations to wear masks in Germany were already in action. Prior 
to the experimental session, written informed consent was obtained from each participant. All 
data were collected anonymously. Each participant was exposed to the complete set of stimuli 
one after another, with the order of stimuli being randomized across participants. Participants 
were asked to spontaneously assess the depicted person’s emotional state from a list of six 
emotions reflecting the same compilation of emotions shown by the different versions of the 
faces (angry, disgusted, fearful, happy, neutral and sad). The personal confidence for each 
assessment had to be indicated on a scale from 1 (very unconfident) to 7 (very confident). There 
was no time limit for giving a response. The general study design (psychophysical testing) was 
given ethical approval by the local ethics committee of the University of Bamberg. The entire 


procedure lasted approximately 20-25 minutes. 


Results 
Data were submitted to further data processing executed by R 4.0.0 (R Core Team, 2014), with 
linear mixed models being analyzed via toolbox /mer (Kuznetsova, Brockhoff, Rune, & 


Christensen). The entire, anonymized, data set is available at the Open Science Framework 





https://osf.io/ka3s6/. 

Overall performance for correctly identifying facial emotions in faces without masks was 
quite remarkable, M = 89.5% (chance rate = 16.7%) with no participant performing below an 
overall rate of 76.4%. As shown by the mean data for each emotional state (Figure 2), presenting 
a mask on faces showed a clear performance drop in reading emotions in faces. With the 
exception of fearful and neutral faces, for which ceiling performance effects were observed, all 


emotional states were harder to read out from faces with masks. 
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Figure 2: Mean percentage of correct assessment of the emotional states for faces with masks (blue) or without masks (red) on 
the face. Error bars indicate confidence intervals CI-95% based on adjusted values for taking within-subjects variances into 
account (Morey, 2008). Asterisks indicate statistical differences between conditions of wearing and non-wearing on basis of 
paired t-tests: ****; p<.0001—ns: not significant. 


We tested the effect of wearing masks on the performance of emotional reading in faces by 
means of linear mixed models (LMM) with face mask (face with a mask vs. without a mask) as a 
fixed factor against a base model (model #0) which only contained the participants and base 
stimuli as random intercepts and face emotion as fixed slopes—FS (fixed factors). We 
furthermore tested in a successive way the effect of the sex and the age group of the face stimuli 
by adding these factors as FS—including all possible interactions of all fixed factors. P-values 


were obtained by likelihood ratio tests of the subsequent models against the respective one-step 
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less complex model. The coefficient of determination for each model was calculated via a 


likelihood-ratio test utilizing the toolbox MuM/n (Barton, 2019). See Table 1 for detailed results. 


Table 1: Linear Mixed Effects analysis of different models in comparison to a simple base model (model #0), separated by the 
two tested dependent variables Yocorrect (percentage of correct emotion classifications) and confidence (for correct emotion 
classifications). The best fitting model, while being parsimonious, is indicated by bold face. FS — fixed slopes (fixed factors); RS — 
random slopes (random factors); df — degrees of freedom; R? —coefficient of determination, based on the likelihood-ratio test; 
PC) — probability of accepting a significant effect despite a non-existent difference regarding the more complex vs. the one-step 
less complex model. 








Dependent variable / | df AIC logLik R? p’) 
tested model 
%correct 
#0: base (random intercepts) | 9 59598 -29790 .090 
#1: + FS face mask | 15 58945 -29458 187 <.0001 
#2: + FS face sex | 27 58850 -29398 .203 <.0001 
#3: + FS face age group | 75 58465 -29157 .266 <.0001 
confidence 
#0: base (random intercepts) | 9 16174 -8078 161 
#1: + FS face mask | 15 15171 -7571 321 <.0001 
#2: + FS face sex | 16 15173 -7570 321 .604 ns 
#3: + FS face age group | 75 15021 -7436 358 <.0001 








Linear Mixed Effects analysis revealed that both dependent variables were impacted by the 
factor face mask. Furthermore, face age group played a role in explaining variance of both 
dependent variables—for face sex, in contrast, we only found an effect for the accuracy of 
emotion reading. 

As face sex as well as face age group were effective in predicting the correctness of 


reading out the emotional state from faces, Figure 3 shows the differentiated data for the three- 
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way interactive effect with face mask. Lower performance in assessing emotions in masked faces 


were found for most emotions and sex and age groups. 


% correct (face sex * face age group) 
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Figure 3: Mean percentage of correctly assessing the emotional states with masks (blue) or without masks (red) on the face, split 
by face sex and face age group. Error bars indicate confidence intervals CI-95% based on adjusted values for taking within- 
subjects variances into account (Morey, 2008). Asterisks indicate statistical differences between conditions of wearing and non- 
wearing on basis of paired t-tests: *: p<.05, **: p<.01, ***: p<.001, ****: p<.0001—ns: not significant. 


Based on the finally selected models with face mask, face sex and face age group being 
included in terms of fixed slopes and their interactions, we obtained several effects of small, 
medium as well as large size (Table 2). Most importantly, regarding the major question of the 
study, face mask had a medium-sized effect on the performance of assessing the emotional state 


of a face and a large-sized effect on the confidence of one’s own assessment (for correct emotion 


classifications). 
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Table 2: Statistics of all involved fixed effects terms of the Linear Mixed Effects analysis for the final models (model #3), 
separated by the two tested dependent variables Ycorrect (percentage of correct emotion classifications) and confidence (for 
correct emotion classifications). k(par)— number of parameters; Cohen’s f—effect size including qualification as small, medium 
or large according to (Cohen, 1988)—smaller effects are not further qualified. Note: abbreviated notations for the terms were 


used to safe space, emotion = face emotion, mask = face mask, sex = face sex, age = face age group. 





% correct confidence 
term k(par) Cohen’s f Cohen’s f 
1 | emotion 5 0.304 medium 0.263 medium 
2 | mask 1 0.253 medium 0.458 large 
3 | sex 1 0.002 0.015 
4 | age 2 0.017 0.045 
5 | emotion:mask 5 0.263 medium 0.204 small 
6 | emotion:sex 5 0.122 small 0.060 
7 | mask:sex 1 0.062 0.002 
8 | emotion:age 10 0.193 small 0.159 small 
9 | mask:age 2 0.019 0.045 
10 | sex:age 2 0.012 0.037 
11 | emotion:mask:sex 5 0.059 0.055 
12 emotion:mask:age 10 0.061 0.054 
13 | emotion:sex:age 10 0.150 small 0.095 
14 | mask:sex:age 2 0.047 0.032 
15 | emotion:mask:sex:age 10 0.137 small 0.096 
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As shown in Figure 4, the confidence data showed a similar but not identical results 
pattern compared to the percentage of correct assessment data in Figure 2. Interestingly, 
confidence data reflected the impact of a face mask emotion reading even more clearly. For 
confidence ratings, also fearful and neutral faces were impacted, probably due to a lack of ceiling 


effects. 
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Figure 4: Mean confidence of assessing the emotional states (for correct classifications) with masks (blue) or without masks 
(red) on the face. Error bars indicate confidence intervals CI-95% based on adjusted values for taking within-subjects variances 
into account (Morey, 2008). Asterisks indicate statistical differences between conditions of wearing and non-wearing on basis of 
paired t-tests: *: p<.05, ****: p<.0001—ns: not significant. 


A drop in performance in reading out emotional states of faces with masks can somehow be 
expected as hardly any visual information of the lower half of the face is available anymore. To 


understand how the lack of information is dealt with, it is important to look at the specific 
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confusion of individual emotional states—when and in which way are emotions misinterpreted 
when face masks are worn? 

In order to learn about these misinterpretations, we generated confusion matrices for the 
viewing conditions with faces without masks and with masks (see Figure 5). When faces were 
shown without masks, the accuracy was much higher as is indicated by clear matches between 
expressed and perceived emotions. With the exception of the emotional state sad, accuracy was 
above 83%, but especially sad was often confused with disgusted (20.3% of the cases). As soon 
as we applied masks to the faces, this overall very high performance broke down dramatically 
and characteristic confusions became apparent. For instance, all emotional states with the 
exception of fearful were repeatedly confused with a neutral state. Sad was often confused with 
disgusted and neutral, and angry was confused with disgusted, neutral and sad. Most drastically 
was the misinterpretation of disgusted as angry, which showed up in nearly 38% of the cases, 


although such a confusion did only happen in 2% of the cases when no face mask was used. 
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Confusion matrix of emotions 
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Figure 5: Confusion matrix of expressed and perceived emotions. Top matrix: faces without masks, bottom matrix: faces with a 
mask. Percentages compile up to 100% for each expressed emotion. The deeper blue the cell, the higher the score of this cell. 


The statistics on the confusion of emotions show clearly how ambiguous an emotional state 


becomes when an ordinary face mask is worn. 


Discussion 

Wearing face masks, even very simple home-made models, is an important measure to 
effectively decrease the chance of transmitting respiratory diseases (van der Sande, Teunis, & 
Sabel, 2008), as is also suggested by the analysis of past pandemics such as the 1918 flu 
pandemic caused by the HIN1 influenza (Bootsma & Ferguson, 2007). People in countries 
where face masks have not been widely used in the past, may still be ambivalent about wearing 
them. Acquaintance is low, wearing a mask when surrounded by too many non-wearers may feel 


strange (Carbon, 2020); and for many, there are obvious handling problems and ergonomic 
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issues including changed air flow characteristics. Yet, the usage of masks is just becoming an 
everyday practice here. 

In the present experiment, we tested the impact of face masks on emotion reading, which 
may have important implications for everyday social interaction. We confronted participants 
with faces showing six different emotions (angry, disgusted, fearful, happy, neutral and sad). 
The results indicate, that emotion recognition was strongly reduced with the exception of fearful 
and neutral faces. For fearful faces, as shown before in the literature (but see Bombari et al., 
2013; Kret & de Gelder, 2012; Wegrzyn, Vogt, Kireclioglu, Schneider, & Kissler, 2017), the 
eyes region, which was not occluded by the mask, provides most of the emotional information 
indicative for this emotional state. For neutral faces, the results have to be interpreted in a 
completely different and cautious way: Although performance for recognizing a neutral state was 
not directly decreased, many emotional states such as happy, sad and angry were misinterpreted 
as neutral, so the genuine emotional state was not perceived anymore. Other emotions such as 
disgusted were confused with angry, and this qualitative misinterpretation which is quite 
impactful (a person who just does not feel aversion to a very specific thing in a certain situation 
and who expresses this spontaneously might be interpreted as an angry, potentially aggressive, 
person) was found in more than one third of all assessments of disgusted faces wearing a mask. 

To further qualify these effects, we have to make clear that the face stimuli originated 
from a scientific database which is aiming to show emotions maximally clear and very 
pronounced. These requirements were nearly perfectly achieved when we look at the very high 
performance data for the original faces without masks. There was hardly any confusion of 
different emotional states (with the exception of sad faces which already showed substantial 


confounds with disgusted at a level of one fifth of the cases). Such a high performance is hardly 
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achievable in everyday life when faces are inspected that show much lower degrees of 
expressing their emotions by the face. Furthermore, in an everyday life scene we will typically 
show lower amounts of attention and will invest less time to inspect the face of a counterpart. 
This means that in natural contexts the impact of face masks on reading emotions could even be 
stronger. It could further be intensified with increased age: As the results of some empirical 
studies indicate, older adults have more difficulties recognizing some of the basic emotions (e.g., 
disgust, happiness, and fear), and even intense problems in recognizing other basic emotions 
such as anger and sadness (Ruffman, Henry, Livingstone, & Phillips, 2008). 

Face masks may complicate social interaction as they disturb emotion reading from facial 
expression. This should, however, not be taken as a reason or an excuse for not wearing masks in 
situations where they are of medical use. We should not forget that humans possess a variety of 
means to interpret another’s state of mind including another’s emotional states. Facial expression 
are not our one and only source of information; we can also take recourse to body posture and 
body language to infer emotional states of our counterpart. The voice characteristic adds 
indications from another modality (Golan, Baron-Cohen, & Hill, 2006), and the social context 
will provide further information (Mondloch, 2012). Direct verbal communication even helps to 
understand the very fine-grained state of a mind. We have options—and it is essential to make 
use of them, not only when being the receiver of socially relevant information, but also when 
being the sender. Emphasizing alternative communicative channels, we can provide sufficient 
information to keep going social interaction in a different, yet effective way. 
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